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This study evaluated the impact of the Student Success Skills (SSS) classroom curriculum delivered in a 
naturalistic setting on the metacognitive functioning of 2,725 middle and high school students in Kentucky. 
SSS was implemented as one intervention to fulfill an Elementary and Secondary School Counseling Grant. 
Results in students' self-reports indicated that those who received the intervention demonstrated increased 
ability to regulate their levels of emotional arousal. No additional significant differences were found. These 
findings differ from the results of previous outcome studies involving SSS. Implications for implementing 
SSS in naturalistic school settings and directions for future research are discussed. 

Keywords: Student Success Skills, naturalistic, metacognitive functioning, classroom curriculum, emotional 
arousal 

The purpose of this study was to evaluate the impact of the Student Success Skills (SSS) school 
counseling curriculum (Brigman, Campbell, & Webb, 2004; Brigman & Webb, 2012) delivered in a 
naturalistic setting on students' metacognitive functioning. In this case, the authors use the term 
naturalistic setting to describe a typical school environment, one which lacks the additional supports 
(e.g., hiring national trainers) that would be present in a more controlled research study. SSS is an 
evidence-based, school counselor-delivered, social-emotional learning intervention that is designed to 
support students by teaching them three integral skill sets: (a) cognitive and metacognitive skills (e.g., 
goal setting, progress monitoring and memory skills); (b) social skills (e.g., interpersonal skills, social 
problem solving, listening and teamwork skills); and (c) self-regulation skills (e.g., managing attention, 
motivation and anger). Research has identified this curriculum as important in promoting students' 
academic achievement and success in school (Collaborative for Academic, Social, and Emotional 
Learning, 2015; Webb & Brigman, 2006). 

SSS was designed based on reviews of educational psychology research literature that identified 
critical skills such as information processing, emotional self-management, and positive social skills 
needed for student success (Bransford, Brown, & Cocking, 1999; Durlak, Weissberg, Dymnicki, Taylor, 
& Schellinger, 2011; Greenberg et al., 2003; Hattie, Biggs, & Purdie, 1996; Marzano, Pickering, & 

Pollock, 2001; Masten & Coatsworth, 1998; Wang, Haertel, & Walberg, 1994; Zins, Weissberg, Wang, 

& Walberg, 2004). The curriculum (Brigman & Webb, 2012) can be delivered in two formats: (1) the 
SSS classroom program and (2) the SSS small group program (Brigman et al., 2004), both of which are 
intended for use with students in grades 4 through 12. SSS is a highly structured, manualized program 
that consists of weekly 45-minute lessons. The classroom format includes five lessons, while the small 
group program includes eight lessons. Both sets of weekly lessons are intended to be delivered in 
chronological order over the corresponding number of consecutive weeks. A 45-minute booster session 
is delivered once a month for 3 months in the spring. 

Developers of SSS designed the curriculum to follow a scripted, manualized format; implementers 
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are encouraged to follow the sequencing, format and language provided in order to ensure fidelity 
of treatment. If practitioners go "off-script" or change the recommended delivery of the lessons, it 
may result in less favorable outcomes that might not meet the same levels as have been found in past 
research. That being said, the SSS program comes with detailed manuals that include recommended 
verbiage, descriptive diagrams and supplemental handouts (Brigman & Webb, 2012). During each 
lesson, students learn and practice strategies in five distinct areas: (a) goal setting, progress monitoring 
and success sharing; (b) creating a caring, supportive and encouraging classroom environment; 

(c) cognitive and memory skills; (d) calming skills; and (e) building healthy optimism. Specific 
strategies are taught and practiced each week and are reviewed and reinforced during subsequent 
lessons. Between lessons, students are encouraged to apply the new strategies that were taught and to 
work on the academic and personal goals that they set for themselves during the SSS session. Teachers 
also are expected to cue students to use the strategies during regular classroom lesson time. 

Research has established the effectiveness of SSS in several quasi-experimental and experimental 
studies. SSS implementation has resulted in enhanced academic achievement as measured by 
standardized achievement tests (Brigman & Campbell, 2003; Brigman, Webb, & Campbell, 2007; 

C. Campbell & Brigman, 2005; Webb, Brigman, & Campbell, 2005) and district math and reading 
achievement measures (Lemberger, Selig, Bowers, & Rogers, 2015). Two studies have suggested that 
the effects of SSS on academic achievement are at least partially mediated by changes in students' 
metacognitive functioning (Lemberger & Clemens, 2012; Lemberger et al., 2015). Lemberger and 
Clemens (2012) found that participation in SSS small groups was associated with improvements in 
students' executive functions (as measured by the Behavior Rating Inventory of Executive Function 
[BRIEF]; Gioia, Isquith, Guy, & Kenworthy, 2000) and increased metacognitive activity (as measured by 
the Junior Metacognitive Awareness Inventory [Jr. MAI]; Sperling, Howard, Miller, & Murphy, 2002). 
Lemberger et al. (2015) found that participation in the classroom version of SSS was associated with 
improvement in executive functions (as measured by the BRIEF-SR; Guy, Isquith, & Gioia, 2004). 

While researchers have established the efficacy of SSS in well-controlled research environments, 
little is known about its effectiveness when delivered in naturalistic settings. The purpose of the present 
study was to measure the effectiveness of the SSS curriculum when delivered in a naturalistic setting 
within regularly functioning schools. In this study, SSS was implemented in five schools in a district in 
Kentucky as part of a school counseling improvement project funded by an Elementary and Secondary 
School Counseling Demonstration Grant awarded by the U.S. Department of Education. The five 
middle and high schools collaborated together through a regional educational cooperative to apply 
for the grant. Demographic information for each school can be found in the Setting section below. The 
grant funded all necessary SSS curriculum materials and provided support for school counselors in 
evaluating the impact of the program. However, funding was not available to hire national trainers. 
National trainers are not a requirement when purchasing a manualized school intervention and many 
schools do not possess the funds needed to hire national trainers. Thus, this funded project provided 
an opportunity to evaluate the effectiveness of SSS in a naturalistic school setting. 

The primary evaluation question was: When implemented in a naturalistic setting, does SSS impact 
students' metacognitive functioning, as determined by (1) knowledge and regulation of cognition as 
measured by the Junior Metacognitive Awareness Inventory (Jr. MAI; Sperling et al., 2002) and (2) 
use of skills related to self-direction of learning, support of classmates' learning, and self-regulation 
of arousal as measured by the Student Engagement in School Success Skills survey (SESSS; Carey, 
Brigman, Webb, Villares, & Harrington, 2013)? The secondary question was: Does the magnitude of 
any changes in metacognitive functioning depend on the degree to which SSS was implemented with 
fidelity? 
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Method 

Setting 

Five participating schools were chosen due to their participation in the Elementary and Secondary 
School Counseling grant. Specific population and demographic data related to all five schools can be 
found in Table 1. 

Table 1 

Brief Description of the Five Schools in the Study 


School 

Enrollment 

& Grades 
Served 

Location 

Ethnicity Rates 

Gender Rates 

Socio-Economic Indicators 

School 1 

1,484 Students 
Grades 9-12 

Suburban/Rural 

African American-0.8% 
American Indian-0% 

Asian American-1.2% 
European American-95.8% 
Hispanic American-0.9% 
Two or More Races-1.2% 

Female-50.2% 
Male-49.8% 

Qualifying for Free 

Lunch-28% 

Qualifying for Reduced 

Lunch-7.6% 

School 2 

1,154 Students 
Grades 6-8 

Suburban/Rural 

African American-1.1% 
American Indian-0.1% 
Asian American-0.7% 
European American-94.1% 
Hispanic American-2% 

Two or More Races-2% 

Female-49% 

Male-51% 

Qualifying for Free 

Lunch-33.5% 

Qualifying for Reduced 

Lunch-7.8% 

School 3 

325 Students 
Grades 7-12 

Urban 

African American-2.8% 
American Indian-0% 

Asian American-0.6% 
European American-95.1% 
Hispanic American-1.2% 
Two or More Races-0.3% 

Female-54.5% 
Male-45.5% 

Qualifying for Free 

Lunch-68.6% 

Qualifying for Reduced 

Lunch-8.6% 

School 4 

570 Students 
Grades 6-8 

Rural 

African American-1.4% 
American Indian-0% 

Asian American-0.2% 
European American-97.5% 
Hispanic American-0.5% 
Two or More Races-0.4% 

Female-48.4% 
Male-51.6% 

Qualifying for Free 

Lunch-50.5% 

Qualifying for Reduced 

Lunch-7.9% 

School 5 

471 Students 
Grades 6-8 

Urban 

African American-11.5% 
American Indian-0.4% 
Asian American-0.4% 
European American-69% 
Hispanic American-10% 
Two or More Races-7.9% 

Female-45.2% 
Male-54.8% 

Qualifying for Free 

Lunch-62.2% 

Qualifying for Reduced 

Lunch-6.4% 


Note: The demographic categories are listed as reported by the state reporting system. 


Participants 

A total of 2,725 students participated in the study with roughly equal numbers of male (50.1%) 
and female (49.9%) students. A relatively large percentage of participants (41.2%) qualified for free or 
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reduced lunch. Less than 1% of participants were classified as Limited English Proficient and 11.2% 
qualified for special education services. In terms of racial and ethnic diversity, the participants were 
less diverse than desired. Eighty-five percent of participating students identified as White (non- 
Hispanic), 4% as Multiracial, 6% as African American, and 4% as Hispanic. All other groups combined 
accounted for the remaining 1% of participants. Proportionally more sixth-, seventh-, and eighth-grade 
students participated in the study. The percentages of participants by grade were: 20.5% sixth grade; 
20.9% seventh grade; 20% eighth grade; 10.7% ninth grade; 9.9% tenth grade; 9.2% eleventh grade; and 
8.7% twelfth grade. 

Completed Jr. MAI surveys (pretest and posttest) were obtained from 1,565 students (57% of 
the total). Completed SESSS surveys were obtained from 1,612 students (59% of the total). School 
counselors were required by the grant to serve as point persons for the delivery and collection of 
pre- and post-instruments. School counselors also were trained on instrument collection using a 
structured, scripted protocol. Instruments were administrated in paper format. Although trained in 
data collection procedures, not all participating school counselors successfully captured both pre- and 
post-assessments. Issues around successfully collecting paper instruments contributed to the loss 
of data. For example, 4% of the Jr. MAI surveys were incomplete and 4% of the SESSS surveys were 
incomplete. Rather than estimating missing data in these instances, it was determined that only data 
from fully complete instruments would be used in the analyses. 

Preparation of the School Counselors 

An SSS manual was provided to each of the schools the year prior to implementation of the 
program. School counselors reviewed the SSS materials and met with the grant project manager 
to discuss the content and instructional processes associated with the SSS classroom guidance 
interventions. Again, school counselors within the five schools did not receive formal training 
from the national SSS trainers on how to implement the curriculum. Formal training was a cost not 
included in the grant. The lack of formal training reflected the more naturalistic approach to the 
study. More often than not, school counselors that purchase a manualized program do not have the 
funding to hire national trainers to guide implementation. School counselors review the manual and 
follow the manualized program during implementation. This approach was reflected in this study. 
School counselors at each school used the SSS curriculum manuals and did their best to adhere to 
the recommended lesson sequence and scripts (Brigman & Webb, 2012). Next, the school counselors 
at each school conducted a pilot SSS small group in the spring semester, prior to the onset of the 
whole-school SSS implementation. The intent of the pilot implementation was to ensure that the 
school counselors were thoroughly familiar with the materials and implementation procedures. 

After conducting the pilot studies, the school counselors provided 3 hours of SSS training to partner 
teachers prior to implementation of the whole-school SSS intervention. The teachers in all schools also 
received a copy of the SSS classroom manual (Brigman & Webb, 2012) to review prior to implementing 
the curriculum with their students. Delivery of the SSS classroom format began the subsequent fall 
semester (2013) and concluded the following spring (2014). 

Procedures for Delivering SSS and Fidelity Issues 

In every school, the school counselors experienced some problems implementing the SSS curriculum 
with fidelity. Implementing the program with complete fidelity would have reflected the exact scope 
and sequence scripted within the SSS manual, mainly delivering the program over a 45-minute 
time period once a week for 5 consecutive weeks. Schools varied from this scope and sequence, 
resulting in a lack of fidelity of the recommended implementation format for SSS. However, schools 
adjusted delivery of SSS in a way that reflected their educational priorities, again reflecting a more 
naturalistic approach than a traditional controlled research study. The primary resistance school 
counselors encountered related to teachers' and administrators' reluctance to lose instructional time 
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as a consequence of in-class implementation of SSS. Each school identified a contextually appropriate 
approach to addressing this initial resistance to devoting class time to SSS. In two of the schools, 
teachers (rather than counselors) delivered the SSS curriculum within their own classrooms. In the 
other three schools, the SSS curriculum was delivered through learning communities (i.e., advisories), 
which are existing scheduled blocks of time during which teachers facilitate small groups (8-15 
students) outside their normal classrooms. For example, a ninth-grade biology teacher might be 
responsible for leading a group of students across several grades with whom the teacher does not 
interact outside of this learning community. The manner in which SSS was delivered in each school is 
detailed below. 

School One. The SSS curriculum was not delivered with complete fidelity. Instead, the school 
leadership determined it was more feasible for teachers to deliver the curriculum through learning 
communities, once a week for 30 minutes for 10 weeks. 

School Two. The SSS curriculum was delivered with reasonable fidelity once a week for 60 minutes 
over a 5-week period by teachers. However, instead of being delivered in a traditional classroom 
format, the school leadership determined it more feasible to deliver the curriculum through learning 
communities. 

School Three. SSS was delivered with reasonable fidelity. Five teachers (trained and supervised 
by the school counselor) delivered the SSS curriculum in the prescribed format detailed in the SSS 
classroom manual. The five teachers delivered SSS to all students in the school through various 
courses, including a study skills course (seventh and eighth grades), a social studies course (ninth 
grade), a biology course (tenth grade), a college readiness course (eleventh grade) and an English 
course (twelfth grade). 

School Four. The SSS curriculum was delivered with reasonable fidelity to all grade levels (sixth 
through eighth) during social studies courses. The social studies teachers were trained and supervised 
by the school counselor to deliver the program. 

School Five. The SSS curriculum was not delivered with complete fidelity. The school leadership 
determined it more feasible to deliver the curriculum through learning communities two to three times 
a week for 25 minutes for each session over a 5-week period. 

Procedures for Collecting SESSS and Jr. MAI Data 

Pretest data were collected at the beginning of the 2013 school year and posttest data were collected 
in late April and early May of the 2013-2014 school year, before the end-of-grade standardized testing 
ensued. Prior to beginning the project, school counselors completed a Collaborative Institutional 
Training Initiative to alert them to issues relating to voluntary participation and confidentiality. 

School counselors were then trained to follow an instrument administration manual (developed for 
the project) so that they could administer the SESSS and the Jr. MAI in a standardized fashion. They 
administered the SESSS and the Jr. MAI using standardized, scripted procedures. In order to protect 
the confidentiality of the students, school counselors changed the student identification numbers for 
each student by adding a randomly determined number for each school. No other person besides 
the school counselor knew the number by which the student identification numbers were changed. 

All data were kept in a locked file cabinet in the primary investigator's office. Data were entered 
into a database, which was saved on an encrypted, password-protected hard drive. As an additional 
safeguard, the data from each school were saved on an external hard drive and transported by hand to 
the primary investigator's office. 
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Instruments 

Junior Metacognitive Awareness Inventory (Jr. MAI) 

The present study used the 18-item version of the Junior Metacognitive Awareness Inventory (Jr. 
MAI; Sperling et al., 2002), a self-report scale that has two subscales that measure students' knowledge 
of cognition and regulation of cognition. The Jr. MAI is used to screen learners for potential metacognitive 
and cognitive strategy interventions. Sperling et al. (2002) developed two versions of the Jr. MAI 
based on the Metacognitive Awareness Inventory (Schraw & Dennison, 1994). The 12-item version 
was developed for students in grades 3 through 5, while the 18-item version was developed for older 
students. 

Available evidence suggests that the Jr. MAI and its subscales are reliable. Sperling et al. (2002) 
reported an internal consistency-based reliability estimate of .82 for the overall scale. Sperling, 
Richmond, Ramsay, and Klapp (2012) reported an internal consistency reliability of .76 for the 
knowledge of cognition subscale and .80 for the regulation of cognition subscale. Sperling et al. 

(2002) found that the Jr. MAI total score (for both versions of the instruments) correlated with other 
direct measures of student metacognition, but not with teachers' ratings of students' metacognitive 
abilities. Relatedly, Sperling et al. (2012) found the student scores on the 18-item version of the Jr. 

MAI correlated significantly with their scores on the Swanson Metacognitive Questionnaire (SMQ; 
Swanson, 1990), their science grade point average and their overall grade point average. Recently, 
the 12-item version of the Jr. MAI was used to measure the impact of SSS on students' metacognitive 
functioning. Lemberger and Clemens (2012) found that SSS delivered in small group format to fourth- 
and fifth-grade students resulted in measurable increases in Jr. MAI scores. 

Student Engagement in School Success Skills (SESSS) Survey 

The study also employed the Student Engagement in School Success Skills (SESSS) survey. The 
SESSS (Carey et al., 2013) is a 27-item scale that was developed to measure the extent to which students 
use strategies that have been shown to be related to enhanced academic achievement (Hattie et al., 

1996; Masten & Coatsworth, 1998; Wang et al., 1994). The SESSS has three subscales that measure 
students' self-direction of learning, support of classmates' learning and self-regulation of arousal. 

Carey et al. (2013) found in an exploratory factor analysis of the SESSS scores of 402 fourth through 
sixth graders that a four-factor solution provided the best model of scale dimensionality considering 
both the solution's clean factor structure and the interpretability of these factors. However, in a 
confirmatory factor analysis study (Brigman et al., 2014) using SESSS scores from a diverse sample 
of almost 4,000 fifth-grade students, researchers found that while a four-factor model fit the data 
well, the scales associated with two subscales correlated so highly (r = .90) as to be indistinguishable. 
Consequently, the items associated with the two factors were combined and the subsequent three- 
factor model also proved to better fit the data. 

Brigman et al. (2014) suggested that the SESSS is best thought of as having three underlying fac¬ 
tors corresponding to self-direction of learning, support of classmates' learning, and self-regulation 
of arousal. Based on factor loadings, Brigman et al. (2014) created three SESSS subscales. The self- 
direction of learning subscale (19 items) reflects the students' intentional use of cognitive and meta¬ 
cognitive strategies to promote their own learning. The support of classmates' learning subscale (six 
items) reflects the students' intentional use of strategies to help classmates learn effectively. Finally, the 
self-regulation of arousal subscale (three items) reflects students' intentional use of strategies to control 
disabling anxiety and cope with stress. 
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Available data indicate that the SESSS is a reliable assessment tool. Carey et al. (2014) reported an 
overall alpha coefficient of 0.91. Furthermore, Villares et al. (2014) reported that the coefficient alphas 
for the three SESSS subscales (self-direction of learning, support of classmates' learning and self¬ 
regulation of arousal) were .89, .79 and .68, respectively. 

Data Analysis 

In order to answer the current study's research questions, the authors conducted separate 
multivariate analysis of variance (MANOVA) with a repeated measure (pretest-posttest time) for the 
Jr. MAI and the SESSS. In the Jr. MAI, the two subscales (knowledge of cognition and regulation of 
cognition) were the dependent variables. For the SESSS MANOVA, the three subscales (self-direction of 
learning, support of classmates' learning, self-regulation of arousal) were the dependent variables. 

After performing the MANOVAs, follow-up repeated measures of analysis of variance (ANOVA) 
were conducted, where appropriate, to determine the significance of the pretest-posttest changes 
for individual subscales. Effect sizes (Cohen, 1988) also were calculated to determine the magnitude 
of pretest-posttest change in subscale associated with the intervention. For significant subscale 
changes, effect sizes were compared across schools to ascertain whether the level of fidelity of SSS 
implementation was related to the intervention's size of effect. 

Results 

MANOVA analyses with a repeated measure (pretest-posttest) were performed to determine 
the differences between Jr. MAI and SESSS subtests across the pretest-posttest time periods in 
order to answer the primary evaluation question. The primary question was: When implemented 
in a naturalistic setting, does SSS impact students' metacognitive functioning, as determined by (1) 
knowledge and regulation of cognition as measured by the Jr. MAI (Sperling et al., 2002) and (2) use 
of skills related to self-direction of learning, support of classmates' learning and self-regulation of 
arousal as measured by the SESSS (Carey et al., 2013)? The results of these MANOVAs are shown in 
Table 2. For the Jr. MAI, the repeated measures MANOVA revealed a significant difference (F (1,1562) 

= 3267.47, p < .001) between the two subscales (knowledge of cognition and regulation of cognition). 
However, no significant difference existed between the pretest and posttest time points. The interaction 
effect of main effects, subscale and time was not significant. 

Table 2 

Repeated Measure MANOVA: Effects of SSS on Students' Metacognitive Activity 


Dependent Measures 

Jr. MAI 

SESSS 

Main effects 

Subtest 

3267.47*** 

356.24*** 

Time 

.3900 

19.84*** 

Interaction effect 

Subtest * Time 

1.4400 

28.25*** 


Note. ***p<001 


In contrast, SESSS repeated measures MANOVA revealed both a significant main effect of Subscale 
(F (2,1610) = 356.24, p < .001) and a significant interaction effect of Subscale x Time (F (2,1610) = 28.25, 
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p < .001). Figure 1 shows that the self-regulation of arousal subscale (subscale 3) corresponded to a 
significantly greater mean change across time, compared to the self-direction of learning (subscale 1) 
and the support of classmates' learning (subscale 2) subscales. 

Figure 1. Pre-Post SSS Treatment Changes in Metacognitive Functioning and Success Skill Use: Means 
for Jr. MAI and SESSS subtests at Pretest (Time 1) and Posttest (Time 2). 

Jr. MAI SESSS 




For Jr. MAI Subtest 1 = Knowledge of Cognition and Subtest 2 = Regulation of Cognition 
For SESSS Subtest 1 = Self-Direction of Learning, Subtest 2 = Support of Classmates' Learning, 
and Subtest 3 = Self-Regulation of Arousal 

Based on these MANOVA results, authors conducted follow-up repeated measures ANOVAs in 
order to test the significance of pretest-posttest changes for all three SESSS subscales. Only the SESSS 
self-regulation of arousal subscale indicated a significant change (F (1, 1610) = 46.147, p < .001) over 
time, reflecting a self-reported increase in students' abilities to regulate their levels of potentially 
debilitating arousal after SSS participation. As shown in Table 3, the effect size of the self-regulation of 
arousal subscale (Cohen's d = -.18) pretest-posttest change would be classified as small (Cohen, 1988). 

Table 3 

Effect Sizes of ANOVAs with Repeated Measures (T1 and T2)for Jr. MAI and SESSS 



Pre 

Post 

Cohen’s d 

Component 

M 

SD 

M 

SD 


Knowledge of Cognition 

3.18 

.83 

3.16 

.85 

+.02 

Regulation of Cognition 

3.96 

.60 

3.96 

.65 

+.00 

The Self-Direction of Learning 

2.32 

.65 

2.33 

.66 

-.02 

The Support of Classmates’ Learning 

2.59 

.74 

2.63 

.74 

-.05 

The Self-Regulation of Arousal 

2.16 

.88 

2.32 

.87 

-.18 
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The secondary research question was: Does the magnitude of any changes in metacognitive 
functioning depend on the degree to which SSS was implemented with fidelity? Implementation 
fidelity was not strongly related to SSS effect size. Cohen's d effect sizes (Cohen, 1988) were computed 
for each school to assess the impact of SSS on self-regulation of arousal. Schools 2, 3 and 4 (who had 
reasonable fidelity of implementation) had effect sizes of .20, .17 and .26 respectively. Schools 1 and 
5 (who had the greatest deviation from implementation fidelity) reported effect sizes of .30 and .13, 
respectively. Therefore, the schools in this study showed considerable variability in effect sizes (.13 
to .30). School differences across other factors (e.g., experience levels of SSS leaders, grade levels of 
students) may have contributed to this variability. 

In summary, although significant findings were not found for pre- and posttests related to the 
Jr. MAI, significant findings were found for the SESSS subscale self-regulation of arousal (p < .001), 
indicating that students increased their ability to regulate levels of potentially debilitating arousal 
after participating in the SSS intervention. Examination of Cohen's d effect sizes suggested that 
implementation fidelity, or the amount that schools varied from the scope and sequence laid out in 
the SSS manuals, did not correlate to level of effect size. This result suggests that the SSS intervention 
resulted in positive outcomes even when practitioners modified the scope and sequence to fit the needs 
of their setting. 

Discussion 

Evaluation Question 1. Does SSS delivered in a naturalistic setting impact students' metacognitive 
functioning? The results of the present study suggest that when implemented in a naturalistic setting, 
SSS can be expected to result in statistically significant increases in students' abilities to regulate 
potentially debilitating emotional arousal. These enhanced abilities might reasonably be expected to 
result in benefits related to improved academic performance (Durlak et al., 2011) and better school 
behavior (perhaps helping students increase their self-control related to daily interpersonal conflict 
and stressful events; Galla & Wood, 2015). The overall effect of SSS on emotional self-regulation, 
while statistically significant, was comparatively small. The present study failed to find evidence that 
SSS influenced other aspects of students' metacognitive functioning, including their knowledge of 
cognition, regulation of cognition, use of strategies related to the self-direction of learning, or use of 
strategies to support fellow classmates' learning. 

Evaluation Question 2. Does the magnitude of any changes in metacognitive functioning depend on the 
degree to which SSS was implemented with fidelity? While the schools in this study showed considerable 
variability in effect sizes, implementation fidelity was not strongly related to SSS effect size. For 
example. School 1 scored lowest on implementation fidelity, but demonstrated the greatest effect 
size (.30). The degree of departure from fidelity was not large enough to detract from SSS's effect on 
students' self-regulation of arousal. 

Relationship to Previous SSS Findings 

The present study failed to replicate the results of previous studies that found significant effects of 
SSS on students' metacognition (Lemberger & Clemens, 2012; Lemberger et al., 2015). While the present 
study as well as Lemberger and Clemens' study (2012) both used the Jr. MAI to measure changes in 
students' metacognition, the two studies differed in terms of SSS delivery format (classroom vs. small 
group). The failure to replicate Jr. MAI-measured changes after SSS participation may be related to 
differences in the format (classroom vs. group) for SSS delivery, or to differences in the delivery context 
(naturalistic vs. controlled). 
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While this study and the Lemberger et al. (2015) study both delivered the SSS classroom format, 
these studies differed in the instruments they used to measure students' metacognitive functioning 
(the SESSS and Jr. MAI vs. the BRIEF). Differences in results may be related to differences in 
instrumentation or to differences in the delivery context (naturalistic vs. controlled). 

Limitations 

A limitation of this study is the use of a one-group pre-post design rather than a quasi-experimental 
design with a control group. Study researchers were constrained by the practical realities of the school 
environment, such that it was not feasible to implement SSS in such a way as to result in a comparison 
group. In addition, researchers were unable to locate similar schools that were willing to have students 
participate in a comparison group. Since it is not always feasible to employ a control group design, a 
one-group pre-post design may be used to attempt to replicate the findings of stronger research studies 
and can point to findings that need to be investigated later using stronger evaluation designs (D. T. 
Campbell & Stanley, 1963; Shadish, Cook & Campbell, 2001). The present evaluation, in fact, furthers 
the understanding of the effects of SSS and highlights directions for future research. 

The failure to collect data in some schools resulted in losses of both the Jr. MAI and the SESSS 
data. Corresponding complete pretest and posttest surveys were obtained from only 57% and 59% 
of participating students, respectively. While the lack of strong control over data collection can be 
thought of as an inherent problem in natural setting evaluations, care should be taken in future studies 
to strengthen data collection processes, as it is difficult to speculate on the impact of the loss of data in 
this study. Loss of data in smaller studies could be extremely harmful to the validity of the study. In 
the present study, there was no reason to believe that the loss of data was related to students' reactivity 
to the intervention; however, it is possible that this loss impacted the results, and additional studies are 
needed to address this issue. 

Implications for Future Research 

It is important to understand the extent and nature of SSS's impact on students' metacognitive 
functioning, whether SSS's well-established impact on academic performance (Brigman & Campbell, 
2003; Brigman, Webb, & Campbell, 2007; C. Campbell & Brigman, 2005; Lemberger et al., 2015; Webb 
et al., 2005) is mediated by changes in students' metacognitive functioning or other variables, and 
whether the impact on academic performance is related to general improvements in all students or 
an improvement in the functions of specific groups of students. Larger-scale intervention studies are 
needed to understand relationships between SSS proximal changes in students' abilities and functioning 
(e.g., metacognitive functioning, emotional self-regulation, engagement, self-efficacy and motivation) 
and distal changes in students' academic performance. It is important to understand which proximal 
changes in students are mediating their distal improvements in academic performance. It could be that 
SSS's effects on academic performance are mediated by one or several of these variables. It also could 
be that SSS's effects on academic performance are mediated by relatively broad variables (e.g., self- 
efficacy) that would be expected to be evident in virtually all students or that are relatively specific (e.g., 
reductions in debilitating test anxiety) and would be expected to be evident in only some students. 

Understanding the mediator(s) of SSS's effects on academic performance would be useful in 
identifying the most appropriate target groups for this intervention. As such, future studies are 
needed to explore whether SSS is most appropriate as a Tier 1 intervention for all students or as a 
Tier 2 intervention for some groups of "at-risk" students. Given that the results of the present study 
suggested that SSS helped students who struggled with self-regulation of arousal, it is especially 
important to examine the effectiveness of SSS as a Tier 2 intervention, specifically for students who 
demonstrate difficulties with emotional self-regulation. Finally, further research is needed to determine 
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the practical significance of SSS on academic performance when it is implemented in less controlled, 
more naturalistic settings, and to determine how deviations from implementation fidelity and other 
contextual factors (e.g., expertise of the SSS leaders) correspond to expected social-emotional and 
academic achievement-related outcomes. 

Implications for Practice 

The results of the present study indicated that SSS, even when implemented in a naturalistic school 
setting (as opposed to a highly controlled setting), can have a positive impact on students' abilities to 
regulate their emotional arousal. The magnitude of the overall impact of SSS on students' ability to 
regulate arousal appears to be relatively small. However, readers should note that this effect size was 
computed based on students in the general population, not students experiencing difficulties with 
emotional self-regulation. It is likely that SSS would have had a larger estimated effect size if the target 
group of participants was those who had emotional self-regulation difficulties. However, the SSS 
curriculum positively impacted student outcomes even when the program was not implemented as 
designed. Though practitioners are encouraged to follow the manual and schedule as recommended, 
the results are encouraging in that impacts can still be found even if practitioners modify the design. 
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